Expose prometheus metrics for ScaledJob resources #4913

yoongon · 2023-08-26T09:37:46Z

Provide a description of what has been changed

Checklist

I have verified that my change is according to the deprecations & breaking changes policy
Changelog has been updated and is aligned with our changelog requirements
A PR is opened to update the documentation on (repo) (if applicable)
Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #4798
Relates to kedacore/keda-docs#1263

github-actions · 2023-08-26T09:38:00Z

Thank you for your contribution! 🙏 We will review your PR as soon as possible.

🏖️ Over the summer, the response time will be longer than usual due to maintainers taking time off so please bear with us.

While you are waiting, make sure to:

Add an entry in our changelog in alphabetical order and link related issue
Update the documentation, if needed
Add unit & e2e tests for your changes
GitHub checks are passing
Is the DCO check failing? Here is how you can fix DCO issues

Learn more about:

Our contribution guide

yoongon · 2023-08-26T09:40:33Z

Hi @JorTurFer
#4817

The previous PR was broken, so I opened this new PR.
I am sorry but I couldn't roll it back.

Please take a look at this again.

Thanks.

pkg/prommetrics/prommetrics.go

pkg/scaling/scale_handler.go

CHANGELOG.md

pkg/scaling/scale_handler.go

zroubalik

Could you please add e2e test for this feature? You can see existing tests for Prom metrics for ScaledObjects: https://github.com/kedacore/keda/blob/main/tests/sequential/prometheus_metrics/prometheus_metrics_test.go

yoongon · 2023-09-25T12:01:43Z

Could you please add e2e test for this feature? You can see existing tests for Prom metrics for ScaledObjects: https://github.com/kedacore/keda/blob/main/tests/sequential/prometheus_metrics/prometheus_metrics_test.go

Hi @zroubalik I will update it together, once the above discussion is done.
Thanks for the suggestion.

zroubalik · 2023-09-26T13:52:39Z

Could you please add e2e test for this feature? You can see existing tests for Prom metrics for ScaledObjects: https://github.com/kedacore/keda/blob/main/tests/sequential/prometheus_metrics/prometheus_metrics_test.go

Hi @zroubalik I will update it together, once the above discussion is done. Thanks for the suggestion.

@yoongon thanks, just FYI we plan to do release tomorrow (Wednesday)

yoongon · 2023-10-31T11:58:59Z

Hi @zroubalik
I updated patch.
Cause I am not familiar with e2e, It takes time. However, I think we can move in parallel.
Could you please check this change first?

BTW, there are some change between this PR and master, I am rebasing it.
I will reupdate it.

Thanks

yoongon · 2024-01-09T11:05:24Z

@zroubalik Thanks for checking. I should have known this one. I closed that PR.
BTW, cause e2e build is not working properly on my local machine, it drags me down.
If you see any missing point on this PR, it will really be appreciated.
I am focusing on e2e now for more debugging.

JorTurFer · 2024-01-11T21:16:40Z

I'm deploying the code into my cluster, I'll give you some feedback about the problem soon

JorTurFer · 2024-01-11T22:43:19Z

You didn't register the metric in Prometheus server:
https://github.com/kedacore/keda/blob/main/pkg/metricscollector/prommetrics.go#L160-L177

That's why now it's not available

JorTurFer · 2024-01-11T22:47:15Z

You probably need to add also a metric record for the jobs here:

keda/pkg/scaling/scale_handler.go

Lines 821 to 822 in 6c1a67d

    
           metrics, isTriggerActive, _, err := cache.GetMetricsAndActivityForScaler(ctx, i, metricSpecs[0].External.Metric.Name) 
        
           if err != nil {

You can do it adding the line metricscollector.RecordScaledJobError(scaledJob.Namespace, scaledJob.Name, err)

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

zroubalik · 2024-01-12T14:41:16Z

/run-e2e sequential
Update: You can check the progress here

yoongon · 2024-01-12T14:50:21Z

@JorTurFer
Thank you so much for the checking.
TBH, I had registered it but i missed after rolled it back a few times. Eventually I forgot.
Thanks again and I've just updated it

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

yoongon · 2024-01-12T15:28:57Z

e2e failed here.

I changed metric name to the original one.
We did a discussion a lot, so I might be confusing to you.

BTW, if you have any concern about the metric name, please let me know again.

JorTurFer · 2024-01-12T15:31:50Z

/run-e2e sequential
Update: You can check the progress here

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

yoongon · 2024-01-13T05:44:08Z

Some e2e errors appear to be related with time.Sleep between creating and deleting wrong scaled resource.
I guess it has been newly added.
I've just added this commit.
44ffcae

JorTurFer · 2024-01-13T11:01:46Z

/run-e2e sequential
Update: You can check the progress here

yoongon · 2024-01-14T09:25:38Z

Looks like final error on e2e. https://github.com//pull/4913/commits/75476df9115d6071234c970e904722577a9460c6

I think this commit will fix it. I must have missed it during the rebase.

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

JorTurFer · 2024-01-14T15:48:33Z

/run-e2e sequential
Update: You can check the progress here

yoongon requested a review from a team as a code owner August 26, 2023 09:37

yoongon force-pushed the feature/prometheus-x branch from a3fdad1 to dddc7e5 Compare August 26, 2023 09:38

yoongon mentioned this pull request Aug 26, 2023

Expose prometheus metrics at ScaledJob like ScaledObject #4817

Closed

4 tasks

yoongon force-pushed the feature/prometheus-x branch from 0fa8998 to 6d91b25 Compare August 26, 2023 11:32

SpiritZhou reviewed Aug 29, 2023

View reviewed changes

pkg/prommetrics/prommetrics.go Outdated Show resolved Hide resolved

SpiritZhou reviewed Aug 29, 2023

View reviewed changes

pkg/scaling/scale_handler.go Outdated Show resolved Hide resolved

SpiritZhou reviewed Aug 29, 2023

View reviewed changes

pkg/scaling/scale_handler.go Show resolved Hide resolved

yoongon force-pushed the feature/prometheus-x branch from 7cc41fc to 32d5ba0 Compare August 29, 2023 12:26

yoongon requested a review from SpiritZhou August 30, 2023 08:46

zroubalik reviewed Sep 6, 2023

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

pkg/scaling/scale_handler.go Show resolved Hide resolved

yoongon force-pushed the feature/prometheus-x branch 2 times, most recently from a24d627 to 84c81dc Compare September 6, 2023 13:01

zroubalik changed the title ~~Expose prometheus metrics at ScaledJob like ScaledObject~~ Expose prometheus metrics for ScaledJob resources Sep 6, 2023

zroubalik reviewed Sep 25, 2023

View reviewed changes

yoongon force-pushed the feature/prometheus-x branch from 07c254e to a431b42 Compare October 31, 2023 11:55

yoongon force-pushed the feature/prometheus-x branch 3 times, most recently from 091abcd to f225e4f Compare November 1, 2023 13:50

zroubalik mentioned this pull request Nov 1, 2023

Add pause metric in prometheus for scaledjob #5140

Closed

yoongon force-pushed the feature/prometheus-x branch 6 times, most recently from af4e86e to aa5d56c Compare November 6, 2023 11:28

yoongon force-pushed the feature/prometheus-x branch from f3bd3ce to d9d0f15 Compare January 12, 2024 14:14

yoongon added 5 commits January 12, 2024 23:14

Expose prometheus metrics at ScaledJob like ScaledObject

db92b32

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

Use assertion for the scaledjob e2e error

01888f4

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

Change to scaledjob for opentelemetry

e51f9ca

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

Resolve conflicts after rebase

55f667c

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

Fix e2e test failing issue

ce7cce6

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

yoongon force-pushed the feature/prometheus-x branch from d9d0f15 to ce7cce6 Compare January 12, 2024 14:14

Fix a duplicated variable

f4a439e

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

yoongon added 2 commits January 13, 2024 00:20

Delete unrelated line at CHANGELOG.md

43bc8e7

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

Use metric name to keda_scaledjob_errors_total

51b3234

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

Add time.Sleep between creating and deleting wrong scaled resources

44ffcae

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

yoongon force-pushed the feature/prometheus-x branch from 8cc29a9 to ececdef Compare January 14, 2024 09:37

Add missed sacledjob error metric value case

75476df

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

yoongon force-pushed the feature/prometheus-x branch from ececdef to 75476df Compare January 14, 2024 09:41

Use original Help message after deciding not to use different label

1f1d9b7

Signed-off-by: Yoon Park <yoongon.park@gmail.com>

zroubalik mentioned this pull request Jan 15, 2024

Release: 2.13.0 #5275

Closed

25 tasks

JorTurFer approved these changes Jan 15, 2024

View reviewed changes

JorTurFer merged commit b220384 into kedacore:main Jan 15, 2024
19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose prometheus metrics for ScaledJob resources #4913

Expose prometheus metrics for ScaledJob resources #4913

yoongon commented Aug 26, 2023 •

edited by zroubalik

Loading

github-actions bot commented Aug 26, 2023

yoongon commented Aug 26, 2023 •

edited

Loading

zroubalik left a comment

yoongon commented Sep 25, 2023

zroubalik commented Sep 26, 2023

yoongon commented Oct 31, 2023 •

edited

Loading

yoongon commented Jan 9, 2024 •

edited

Loading

JorTurFer commented Jan 11, 2024

JorTurFer commented Jan 11, 2024

JorTurFer commented Jan 11, 2024

zroubalik commented Jan 12, 2024 •

edited by github-actions bot

Loading

yoongon commented Jan 12, 2024

yoongon commented Jan 12, 2024

JorTurFer commented Jan 12, 2024 •

edited by github-actions bot

Loading

yoongon commented Jan 13, 2024 •

edited

Loading

JorTurFer commented Jan 13, 2024 •

edited by github-actions bot

Loading

yoongon commented Jan 14, 2024 •

edited

Loading

JorTurFer commented Jan 14, 2024 •

edited by github-actions bot

Loading

Expose prometheus metrics for ScaledJob resources #4913

Expose prometheus metrics for ScaledJob resources #4913

Conversation

yoongon commented Aug 26, 2023 • edited by zroubalik Loading

Checklist

github-actions bot commented Aug 26, 2023

yoongon commented Aug 26, 2023 • edited Loading

zroubalik left a comment

Choose a reason for hiding this comment

yoongon commented Sep 25, 2023

zroubalik commented Sep 26, 2023

yoongon commented Oct 31, 2023 • edited Loading

yoongon commented Jan 9, 2024 • edited Loading

JorTurFer commented Jan 11, 2024

JorTurFer commented Jan 11, 2024

JorTurFer commented Jan 11, 2024

zroubalik commented Jan 12, 2024 • edited by github-actions bot Loading

yoongon commented Jan 12, 2024

yoongon commented Jan 12, 2024

JorTurFer commented Jan 12, 2024 • edited by github-actions bot Loading

yoongon commented Jan 13, 2024 • edited Loading

JorTurFer commented Jan 13, 2024 • edited by github-actions bot Loading

yoongon commented Jan 14, 2024 • edited Loading

JorTurFer commented Jan 14, 2024 • edited by github-actions bot Loading

yoongon commented Aug 26, 2023 •

edited by zroubalik

Loading

yoongon commented Aug 26, 2023 •

edited

Loading

yoongon commented Oct 31, 2023 •

edited

Loading

yoongon commented Jan 9, 2024 •

edited

Loading

zroubalik commented Jan 12, 2024 •

edited by github-actions bot

Loading

JorTurFer commented Jan 12, 2024 •

edited by github-actions bot

Loading

yoongon commented Jan 13, 2024 •

edited

Loading

JorTurFer commented Jan 13, 2024 •

edited by github-actions bot

Loading

yoongon commented Jan 14, 2024 •

edited

Loading

JorTurFer commented Jan 14, 2024 •

edited by github-actions bot

Loading